Moving vehicles, such as aircraft, watercraft, and ground-based vehicles (herein referred to generically as “vehicles”) often must operate in conditions of limited visibility due to rain, fog, glare, darkness, and other environmental conditions that prevent a clear view of the surrounding scene. As used herein, the term “scene” refers to a view of an object or of a volume from a particular point and looking in a particular direction in three-dimensional space. Thus, the “scene” visible to a pilot through a cockpit window will change whenever the attitude of the aircraft changes due to pitch, roll, or yaw and/or whenever the position of the aircraft changes its location in three-dimensional space.
It is particularly important that the person or persons operating the vehicle have a clear enough view of the surrounding scene to operate the vehicle safely, e.g., to avoid a crash, collision, or navigation error. A pilot that is landing an aircraft, for example, must be able to not only determine the location of the runway but also be able to see unexpected (or expected) obstacles in enough time to be able to take evasive action or perform some other maneuver to avoid the obstacles.
One approach to this problem is to provide to the pilot an image of the scene as it is captured by an image sensor or camera (hereinafter referred to generically as a “camera”) and presenting it to the pilot on a video display unit. A slight improvement on this technique is to use a camera that is receptive to frequencies outside of the normal range of human vision, such as the infra-red (IR) or ultraviolet (UV) frequencies, and presenting those images to the pilot. UV light passes through clouds and fog, for example, and IR light is radiated by heat sources. The images produced by IR and UV cameras, however, look very strange to humans, which make IR and UV images more difficult to understand and process than visible light images, which humans are accustomed to seeing.
Yet another improvement is to combine images from multiple cameras or from cameras sensitive to different frequencies by adding the images together in a process referred to herein as “mixing”, in which multiple images are combined according to some ratio. For example, a video mixer may multiply the intensity of one image to 30%, multiply the intensity of another image by 70%, and add the images together to provide an image whose brightest areas have 100% intensity. Conventional mixing techniques may adjust the relative ratios of each image's contribution into the whole, e.g., from 30/70 to 40/60, 80/20, 97/3, or other relative ratio applied to each entire image prior to summing the two images together to provide the output image.
Mixing also has disadvantages. For example, one image may have valuable information but is so bright (or “hot”) that by the time it is scaled down to avoid washing out the output image, the desired detail is also lost.
Another approach is to provide an aircraft pilot with a head-up display (HUD) that projects a synthetic image that is visually overlaid on top of the pilot's normal view, i.e., the view from the cockpit window. The synthetic image may display information about the aircraft's altitude, speed, and heading along with graphic elements that identify, highlight, or outline important features of the scene, such as the location of the runway.
One problem, however, is that even the synthetic images representing features and obstacles are usually generated from image data provided by image sensors. In order for a HUD to display representations of features or obstacles as synthetic graphical objects, image sensors on the vehicle must be able to detect those features and obstacles. If the image sensors cannot detect or distinguish the important features or obstacles, the imagery provided by the sensor on the HUD may be of little value to the pilot.
Accordingly, in view of the disadvantages of conventional vision systems used by aircraft and other types of vehicles, there is a need for improved methods and systems that generate composite images from multiple imaging sensors.